Description

sets up a vllm proxy to a locally running ollama server. configures a connection and runs tests against it.

removes opentelemetry dumps to console
removes progress pushing state to the console both of which were polluting the logs

Added "HOSTED_VLLM_API_BASE" in UI key for vllm support.
https://docs.litellm.ai/docs/providers/vllm#embeddings

TODO

• evaluation is not working yet.
• The evaluation response is empty.
• remove hard-coded model: accept model and API base as input from the config
• fix azure
• fix openai, anthropic and gemini are not working due to config/env params. needs to be debugged and fixed

Steps to run the vllm proxy to ollama locally

• Install ollama and vllm.
• Download the llama3.2:3b model

ollama pull llama3.2:3b

• start the vllm server

cd examples/vllm_server
python vllm_ollama_server.py

Tested by running the command:

compliant-llm test --prompt "You are a helpful assistant" --strategy prompt_injection --provider "hosted_vllm/llama3.2:3b" 

or

compliant-llm test  --config_path  "configs/config.yaml"

or

compliant-llm dashboard

Screenshots/Results

Reference Links

#110

Checklist

This PR includes the following (tick all that apply):

• Tests added/updated for new/changed functionality
• Bug Fix (explain the bug in the description)
• Refactoring or optimizations (no functional changes)
• Documentation updated (README, docstrings, etc.)
• Build or deployment related changes
• Dependency updates (requirements.txt/pyproject.toml)
• Setup change (update README/setup instructions if required)
• Code style checks passed (PEP8, flake8, black, etc.)
• Type hints added/updated (if applicable)
• Pre-commit hooks run (if configured)